Repository navigation
route-resident: support 256-expert top-8 models (Qwen3.6-35B-A3B) - #2321
atomicmilkshake wants to merge 1 commit into
Conversation
Generalize the warp-reduction route-resident kernel and API across both 512/10 and 256/8 (Qwen3.6-35B-A3B) geometries, retaining backward-compatible call sites and symbols. Tail ranks default to 5-7 for 256/8 (6-9 for 512/10) unless overridden by STRATA_ROUTE_RESIDENT_RANKS. Parameterize the bit-for-bit route_resident_parity test across both configurations (240 synthetic cases each, 0 mismatches, 0 ulp). Validated on PR Niko1221#1820 (+9.7% decode / -13.1% window on 0.1.42 with tail skip default 7).
|
Checked on an RTX 5070 Ti (sm_120, Linux, CUDA 13.1): |
|
Thanks for the pull request. We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected. To reopen it takes about two minutes:
The rules are in CONTRIBUTING.md. |
Summary
Generalizes the opt-in \STRATA_ROUTE_RESIDENT\ kernel and verifier guard from 512/10 models to also support 256-expert top-8 models (\qwen35moe\ / Qwen3.6-35B-A3B).
Follows Ethan's warp-reduction rewrite (#1737) and separates out the 256/8 portion previously included in #1820 into a clean, standalone change on top of upstream \main.
Details
Kernel parameterization (\src/kernels/cuda/native_router.cu):
oute_resident_impl<NE, K>\ across warp-multiple expert counts. Both 512 and 256 are multiples of 32, so the __shfl_down_sync\ reduction tree stays convergent across lanes.
oute_resident_k\ (512/10) CUDA symbol and entry point, and instantiates
oute_resident_geometry_k<256, 8>\ for Qwen geometries.
ative_route_resident\ overload so existing 512/10 call sites (\src/core/mtp.cpp, metal stubs) continue without change.
Verifier & Configuration (\src/core/verify.cpp):
Parity Test (\src/kernels/route_resident_parity.cu):
Context
Extracted from #1820 per maintainer single-change guideline. Validated on 0.1.42 (+9.7% decode / -13.1% window on Qwen3.6-35B-A3B).